Skip to content

Weaver live check test util#5088

Open
lmolkova wants to merge 17 commits intoopen-telemetry:mainfrom
lmolkova:weaver-live-check
Open

Weaver live check test util#5088
lmolkova wants to merge 17 commits intoopen-telemetry:mainfrom
lmolkova:weaver-live-check

Conversation

@lmolkova
Copy link
Copy Markdown
Member

@lmolkova lmolkova commented Apr 14, 2026 •
edited
Loading

Description

Adds WeaverLiveCheck helper that allows to validate instrumentation output against otel semconv.

Examples:

  1. you can call report = check.end() and it won't raise. You can explore report (which contains a list of violations and also full weaver report as dict from json) - party on
        with WeaverLiveCheck() as weaver:
            # ... configure provider, emit telemetry ...
            provider.force_flush()
            report = weaver.end()

        seen_metrics = report["statistics"]["seen_registry_metrics"]
        self.assertEqual(seen_metrics.get("test.metric"), 1)
  1. you can call report = check.end_and_check() and it'd raise LiveCheckError (extends AssertionError) if any violations are found. You still get full report as a property on the error if you want to assert on more
        with WeaverLiveCheck() as weaver:
            # ... configure provider, emit telemetry ...
            provider.force_flush()

            with self.assertRaises(LiveCheckError) as cm:
                weaver.end_and_check()

        seen_metrics = cm.exception.report["statistics"]["seen_registry_metrics"]
        self.assertEqual(seen_metrics.get("test.metric"), 1)

Checklist:

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@lmolkova lmolkova requested a review from a team as a code owner April 14, 2026 06:08
xrmx
xrmx approved these changes Apr 14, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@xrmx xrmx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this, thanks a lot!

Comment thread tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py Outdated
Comment thread tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py
xrmx
xrmx reviewed Apr 14, 2026
View reviewed changes
Comment thread .github/workflows/templates/test.yml.j2 Outdated
@lmolkova lmolkova force-pushed the weaver-live-check branch from 80a4f86 to 284dece Compare April 14, 2026 17:28
xrmx
xrmx reviewed Apr 15, 2026
View reviewed changes
Comment thread .github/workflows/templates/test.yml.j2 Outdated
Comment thread tests/opentelemetry-test-utils/test-requirements.txt Outdated
herin049
herin049 reviewed Apr 16, 2026
View reviewed changes
Comment thread tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py Outdated
Comment thread tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py Outdated
Comment on lines +51 to +65
raw: list[dict] = []

def collect(obj: Any) -> None:
if isinstance(obj, dict):
if "live_check_result" in obj:
for advice in obj["live_check_result"].get("all_advice", []):
if advice.get("level") == "violation":
raw.append(advice)
for val in obj.values():
collect(val)
elif isinstance(obj, list):
for item in obj:
collect(item)

collect(report)
Copy link
Copy Markdown
Contributor

@herin049 herin049 Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we've dropped 3.9, you could do:

def _collect(obj: Any) -> list[dict]:
    match obj:
        case {"live_check_result": {"all_advice": advices}, **_rest}:
            violations = [a for a in advices if a.get("level") == "violation"]
            return violations + list(chain.from_iterable(_collect(v) for v in obj.values()))
        case dict():
            return list(chain.from_iterable(_collect(v) for v in obj.values()))
        case list():
            return list(chain.from_iterable(_collect(item) for item in obj))
        case _:
            return []

raw = _collect(report)

Copy link
Copy Markdown
Member Author

@lmolkova lmolkova Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

Copy link
Copy Markdown
Member Author

@lmolkova lmolkova Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it segfaults on pypy 310 on ubuntu, reverted some parts

Fatal Python error: Segmentation fault

Stack (most recent call first, approximate line numbers):
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 54 in _collect
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 65 in <genexpr>
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 54 in _collect
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 69 in <genexpr>
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 54 in _collect
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 65 in <genexpr>
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 54 in _collect
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 41 in _extract_violations
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/src/opentelemetry/test/weaver_live_check.py", line 175 in violations
  File "/opt/hostedtoolcache/PyPy/3.10.16/x64/lib/pypy3.10/functools.py", line 973 in __get__
  File "/home/runner/work/opentelemetry-python/opentelemetry-python/tests/opentelemetry-test-utils/tests/test_weaver_live_check.py", line 64 in test_end_and_check_no_violations

https://github.com/open-telemetry/opentelemetry-python/actions/runs/24545978826/job/71761592153?pr=5088

@xrmx xrmx moved this to Approved PRs in Python PR digest Apr 16, 2026
lmolkova
lmolkova commented Apr 17, 2026
View reviewed changes
Comment thread tests/opentelemetry-test-utils/test-requirements.txt
-e tests/opentelemetry-test-utils
# these are required for weaver integration tests, we're running that only on linux
# because of lack of support for gRPC on Windows in some cases.
# note: tox does not support PEP 508 markers on `-e` editable installs, so these are installed non-editable
Copy link
Copy Markdown
Member Author

@lmolkova lmolkova Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for whatever reason tox does not like -e along with platform marker, but we don't really need those as editable

@lmolkova lmolkova force-pushed the weaver-live-check branch from 39b4fcf to 270c9d2 Compare April 17, 2026 03:21
@lmolkova lmolkova force-pushed the weaver-live-check branch from 1094e5b to 00d6e47 Compare April 17, 2026 04:15
MikeGoldsmith
MikeGoldsmith approved these changes Apr 17, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@MikeGoldsmith MikeGoldsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xrmx xrmx xrmx approved these changes

@herin049 herin049 herin049 approved these changes

@MikeGoldsmith MikeGoldsmith MikeGoldsmith approved these changes

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

Python PR digest
Status: Approved PRs

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@lmolkova @xrmx @MikeGoldsmith @herin049